01. Learning Plan

Learning Plan

The first part of this Nanodegree program covers the basics of reinforcement learning and lasts 4 weeks. For these first 4 weeks of the program, you will build a strong background in reinforcement learning (without neural networks), before spending the remaining time in the course learning how to leverage neural networks to train intelligent agents.

## Week 1

In the first week, you will learn the basics of reinforcement learning.

Lesson: Introduction to RL

In this lesson, you'll explore a friendly introduction to reinforcement learning.

Lesson: The RL Framework: The Problem

In this lesson, you'll learn how to specify a real-world problem as a Markov Decision Process (MDP), so that it can be solved with reinforcement learning.

Lesson: The RL Framework: The Solution

In this lesson, you'll learn all about value functions and optimal policies.

Readings

  • Chapter 1 (especially 1.1-1.4) of the textbook
  • Chapter 3 (especially 3.1-3.3, 3.5-3.6) of the textbook

## Week 2

In the second week, you'll build your own agents to solve the reinforcement learning problem.

Lesson: Dynamic Programming (Optional)

In this lesson, you'll build some intuition for the reinforcement learning problem by learning about a class of solution methods that solve a slightly easier problem. (This lesson is optional and can be accessed in the extracurricular content.)

Lesson: Monte Carlo Methods

In this lesson, you'll learn about a class of solution methods known as Monte Carlo methods. You'll implement your own Blackjack-playing agent in OpenAI Gym.

Readings

  • Chapter 4 (especially 4.1-4.4) of the textbook (This reading is optional and accompanies the optional Dynamic Programming lesson.)
  • Chapter 5 (especially 5.1-5.6) of the textbook

## Week 3

In the third week, you'll leverage a slightly more sophisticated class of solution methods to build your own agents in OpenAI Gym.

Lesson: Temporal-Difference Methods

In this lesson, you'll learn how to apply temporal-difference methods such as SARSA, Q-learning, and Expected SARSA to solve both episodic and continuing tasks.

Lesson: Solve OpenAI Gym's Taxi-v2 Task

In this lesson, you'll apply what you've learned to train a taxi to pick up and drop off passengers.

Readings

## Week 4

In the last week, you will learn how to adapt the algorithms that you've been learning about to solve a larger class of problems.

Lesson: RL in Continuous Spaces

In this lesson, you'll explore how to use techniques such as tile coding and coarse coding to expand the size of the problems that can be solved with traditional reinforcement learning algorithms.

Lesson: What's Next?

In this lesson, you'll learn more about what's to come in the next three parts of the Nanodegree program. You'll also get some tips for how to best spend your time, if you finish this first part of the Nanodegree program early!

Readings

  • Chapter 9 (especially 9.1-9.7) of the textbook